Different tools and methods have been developed for systems design and testing. During the 1980's, structuring tools were developed allowing a system designer to specify the environment of his system and then, following a hierarchical approach, to specify the expected behavior of each part of the system. The structured analysis method is a good illustration of this first period.
However, for complex systems, it became quickly clear that this was not enough and design tools became executable tools allowing simulation. During this paradigm shift however, the link between early system requirements and system executable specification was not preserved at first. Of course, simulation under virtual constraints does not prevent testing in a practical environment and the traceability from early requirements to an executable model and then test cases was not built in. Coding was still made after simulation, allowing detection and correction of mistakes made at the early modeling steps.
Later it became clear that for very large developments, involving many persons, the link between early requirements and systems specification was of prime importance and requirements management systems started to establish the links between requirements, models, pieces of code and tests afterwards. At this step however, which represents the current state of the art, the link between requirements, models and tests is not formal and has to be maintained manually. Approaches like Universal Modeling Language (UML) make an attempt to capture the different process steps (requirements modeling and testing), but the UML formalism is too vague to support really more than an informal link between the different design steps.
In parallel, automated test generation strategies evolved to try to exploit information of executable models, but in the absence of requirements capture in the testing environment, testing is performed blindly with some hazardous approach and it is well known that such testing is very efficient on certain parts of a model but simply omits other parts.
In the case of manually generated tests, the state of the art consists in specifying test vectors that can be executed when the component under test has been set in a proper initial state, something which may be very difficult to reach for black box testing of a series electronic control unit (ECU).
It should be noted that for domains where the hardware design is very optimized, black box testing of series components is always necessary as the embedding of a real upper tester is too expensive.
Also, once a model is designed, a function may be distributed and then comes the challenge of providing unitary testing for each distributed piece of function; especially when the different pieces have to be developed by different suppliers. This is an open issue. At present, the favored approach is to wait for the different component prototypes, make the embodiment and then interact and debug. This means that real debugging starts when the hardware design is frozen, a hardly efficient process.
In case of a fault-tolerant function, it is also required to include fault injection in testing and combine this with the distributed character of such functions. At this stage again, no standard and determinist approach is available.
In U.S. Pat. No. 5,394,347, a method of designing a testing environment is disclosed which employs an EFSM (Extended Finite State Machine). There is no distinction between a user request and a system response in an EFSM and all user requests and system responses to those requests are always merged into transitions. In this proposal, system events are inputs but also cover outputs, such as for example a bell ringing. It is a disadvantage of U.S. Pat. No. 5,394,347 that the technology used does not distinguish between user requests and system responses, because it is then necessary to keep track of at least a “Variable Stack” and “Model Stack”. Furthermore, this also means that in U.S. Pat. No. 5,394,347 it is only possible to validate the correct implementation of a specification and it is not possible to validate the specification itself. The simulations performed in accordance with the proposal in U.S. Pat. No. 5,394,347 are not performed in real time because time is considered as a variable of the environment of the system being tested that can be set with predefined functions referred to in the specification as “Force”. This mechanism is not sufficient for the simulation of signal processing.
It can therefore be seen that there is a continuing need for improved methods for a uniform design of system testing which allows validating faithfully the system model and its physical representations at the different steps of its realization, especially in the case of distributed functions design.